Concept for an Algorithm Testing and Evaluation Program at NIST

F i g u re 1 Coord ina t e measu r ing system—data analysis software analyzes coordinate data, producing surface fits. "any piece of equipment which collects three dimensional coordinates (points), calculates and displays additional information using the measured points." [1]. Figure 1 displays a high level view of how a CMS works. A CMS—such as a coordinate measurement machine, vision system, or theodolite—works by measuring points on the part surface and then analyzing the points to evaluate the part. The coordinate data are evaluated by data analysis software embedded in the CMS. The data analysis software is what produces the dimensional measurements—surface fits. CMSs are widely used by the dimensional metrology community during the inspection of manufactured parts. The data analysis software within CMSs is becoming increasingly important. During the last few years research [2] shows that data analysis software contributes significantly to the total measurement error (measurement uncertainty) of a CMS. Hence, the overall performance of CMSs is affected. The performance of data analysis software is affected by the choice of analysis method, quality of the software, and characteristics of the specific measurement tasks. There are no standards for evaluating these and other effects of software on the overall uncertainty of measurement. Through standard efforts by the dimensional metrology community, industry has identified the need for a formal mechanism to test and evaluate data analysis software within CMSs. This report presents a concept for a formal mechanism, the NIST Algorithm Testing and Evaluation Program (ATEP). Section 2 describes the current problem with CMS data analysis software. Section 3 presents a proposed solution to the problem. Section 4 explains the technical approach to the proposed solution. Section 5 presents a summary.